Skip to content

fix: retry account bootstrap with modern database relation to accomodate pg_hba not being ready yet#81

Open
jansdhillon wants to merge 2 commits intocanonical:mainfrom
jansdhillon:fix-first-admin-new-pg
Open

fix: retry account bootstrap with modern database relation to accomodate pg_hba not being ready yet#81
jansdhillon wants to merge 2 commits intocanonical:mainfrom
jansdhillon:fix-first-admin-new-pg

Conversation

@jansdhillon
Copy link
Copy Markdown
Contributor

@jansdhillon jansdhillon commented Mar 6, 2026

Fixes https://warthogs.atlassian.net/browse/LNDENG-4035

make deploy

Run the new integration test:

LANDSCAPE_CHARM_USE_HOST_JUJU_MODEL=1 uv run pytest tests/integration/test_bundle.py::test_bootstrap_account_created_with_modern_database

The bundle has been modified to bootstrap the first admin account:

admin_email: john@example.com
admin_name: john
admin_password: pwd

Using the debug log, observe the behavior when the modern database relation joins:

juju debug-log --replay --include landscape-server/0 -m landscape-server-operator-build --tail

...

unit-landscape-server-0: 01:34:25 INFO unit.landscape-server/0.juju-log Fixing python paths
unit-landscape-server-0: 01:34:25 INFO unit.landscape-server/0.juju-log Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:34:25 INFO unit.landscape-server/0.juju-log ['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'REDACTED', '--root_url', 'https://landscape.local/']
unit-landscape-server-0: 01:34:25 INFO unit.landscape-server/0.juju-log Fixing python paths
unit-landscape-server-0: 01:34:34 ERROR unit.landscape-server/0.juju-log Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 161, in wrap_exceptions
    yield
  File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 437, in raw_connect
    return self._raw_connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 418, in _raw_connect
    raw_connection = ConnectionWrapper(psycopg2.connect(self._dsn), self)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/canonical/landscape/bootstrap-account", line 6, in <module>
    canonical.landscape.scripts.bootstrap_account.run()
  File "/opt/canonical/landscape/canonical/landscape/scripts/bootstrap_account.py", line 80, in run
    run_bootstrap_account(sys.argv[1:])
  File "/opt/canonical/landscape/canonical/landscape/scripts/bootstrap_account.py", line 65, in run_bootstrap_account
    bootstrap_account_helper(
  File "/opt/canonical/landscape/canonical/landscape/api/account.py", line 733, in bootstrap_account_helper
    account = get_account_by_name("standalone")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/canonical/landscape/canonical/landscape/model/main/account.py", line 1683, in get_account_by_name
    collection = AccountCollection(get_main_store(), account_names=[name])
                                   ^^^^^^^^^^^^^^^^
  File "/opt/canonical/landscape/canonical/landscape/model/main/store.py", line 19, in get_main_store
    return get_store_by_name("main", isolation_level=isolation_level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/canonical/landscape/canonical/landscape/model/store.py", line 103, in get_store_by_name
    store = zstorm.get(name)
            ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/zope/zstorm.py", line 176, in get
    return self.create(name, default_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/zope/zstorm.py", line 152, in create
    store = Store(database)
            ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/store.py", line 79, in __init__
    self._connection = database.connect(self._event)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/database.py", line 576, in connect
    return self.connection_factory(self, event)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/database.py", line 262, in __init__
    self._raw_connection = self._database.raw_connect()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 436, in raw_connect
    with wrap_exceptions(self):
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 175, in wrap_exceptions
    raise wrapped.with_traceback(tb) from e
  File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 161, in wrap_exceptions
    yield
  File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 437, in raw_connect
    return self._raw_connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 418, in _raw_connect
    raw_connection = ConnectionWrapper(psycopg2.connect(self._dsn), self)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
storm.database.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?


unit-landscape-server-0: 01:34:34 ERROR unit.landscape-server/0.juju-log Cannot modify autoregistration because no account exists.
unit-landscape-server-0: 01:34:34 INFO unit.landscape-server/0.juju-log Generating new random secret token
unit-landscape-server-0: 01:34:34 INFO unit.landscape-server/0.juju-log Generating new random cookie encryption key
unit-landscape-server-0: 01:34:34 INFO unit.landscape-server/0.juju-log Writing secret token
unit-landscape-server-0: 01:34:34 INFO unit.landscape-server/0.juju-log Fixing python paths
unit-landscape-server-0: 01:34:35 INFO unit.landscape-server/0.juju-log Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:34:35 INFO unit.landscape-server/0.juju-log Writing cookie encryption key
unit-landscape-server-0: 01:34:35 INFO unit.landscape-server/0.juju-log Fixing python paths
unit-landscape-server-0: 01:34:35 INFO unit.landscape-server/0.juju-log Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:34:37 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-landscape-server-0: 01:34:37 INFO juju.worker.uniter found queued "start" hook
unit-landscape-server-0: 01:34:38 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-landscape-server-0: 01:34:40 INFO juju.worker.uniter.operation ran "load-balancer-certificates-relation-changed" hook (via hook dispatching script: dispatch)
unit-landscape-server-0: 01:34:40 INFO unit.landscape-server/0.juju-log database:7: database created at 2026-03-06 08:34:40.517712
unit-landscape-server-0: 01:34:40 INFO unit.landscape-server/0.juju-log database:7: New database endpoint is 10.1.77.31:5432
unit-landscape-server-0: 01:34:40 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:34:40 INFO unit.landscape-server/0.juju-log database:7: Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:34:40 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:34:49 INFO unit.landscape-server/0.juju-log database:7: ['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'REDACTED', '--root_url', 'https://landscape.local/']
unit-landscape-server-0: 01:34:49 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:34:55 ERROR unit.landscape-server/0.juju-log database:7: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1806, in <module>
    main(LandscapeServerCharm)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 502, in main
    manager.run()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 486, in run
    self._emit()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 421, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 465, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 3625, in _on_relation_changed_event
    getattr(self.on, "database_created").emit(
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 952, in _database_relation_changed
    if not self._migrate_schema_bootstrap(roles.owner):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1038, in _migrate_schema_bootstrap
    self._bootstrap_account()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1567, in _bootstrap_account
    raise CalledProcessError(
subprocess.CalledProcessError: Command '['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'pwd', '--root_url', 'https://landscape.local/']' returned non-zero exit status 1.
unit-landscape-server-0: 01:34:55 ERROR juju.worker.uniter.operation hook "database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-landscape-server-0: 01:34:55 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-landscape-server-0: 01:35:00 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-landscape-server-0: 01:35:00 INFO unit.landscape-server/0.juju-log database:7: database created at 2026-03-06 08:35:00.962602
unit-landscape-server-0: 01:35:01 INFO unit.landscape-server/0.juju-log database:7: New database endpoint is 10.1.77.31:5432
unit-landscape-server-0: 01:35:01 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:01 INFO unit.landscape-server/0.juju-log database:7: Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:35:01 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:07 INFO unit.landscape-server/0.juju-log database:7: ['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'REDACTED', '--root_url', 'https://landscape.local/']
unit-landscape-server-0: 01:35:07 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:23 ERROR unit.landscape-server/0.juju-log database:7: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1806, in <module>
    main(LandscapeServerCharm)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 502, in main
    manager.run()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 486, in run
    self._emit()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 421, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 465, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 3625, in _on_relation_changed_event
    getattr(self.on, "database_created").emit(
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 952, in _database_relation_changed
    if not self._migrate_schema_bootstrap(roles.owner):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1038, in _migrate_schema_bootstrap
    self._bootstrap_account()
  File "/var/lib/juju/agents/unit-landscape-server-0/charm/src/charm.py", line 1567, in _bootstrap_account
    raise CalledProcessError(
subprocess.CalledProcessError: Command '['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'pwd', '--root_url', 'https://landscape.local/']' returned non-zero exit status 1.
unit-landscape-server-0: 01:35:23 ERROR juju.worker.uniter.operation hook "database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-landscape-server-0: 01:35:23 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-landscape-server-0: 01:35:34 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-landscape-server-0: 01:35:34 INFO unit.landscape-server/0.juju-log database:7: database created at 2026-03-06 08:35:34.711301
unit-landscape-server-0: 01:35:34 INFO unit.landscape-server/0.juju-log database:7: New database endpoint is 10.1.77.31:5432
unit-landscape-server-0: 01:35:34 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:35 INFO unit.landscape-server/0.juju-log database:7: Migrated service.conf: CompletedProcess(args=['/opt/canonical/landscape/migrate-service-conf'], returncode=0, stdout='Updated config file at /etc/landscape/service.conf\n', stderr='')
unit-landscape-server-0: 01:35:35 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:38 INFO unit.landscape-server/0.juju-log database:7: ['/opt/canonical/landscape/bootstrap-account', '--admin_email', 'john@example.com', '--admin_name', 'john', '--admin_password', 'REDACTED', '--root_url', 'https://landscape.local/']
unit-landscape-server-0: 01:35:38 INFO unit.landscape-server/0.juju-log database:7: Fixing python paths
unit-landscape-server-0: 01:35:41 INFO unit.landscape-server/0.juju-log database:7: Admin account successfully bootstrapped!

It will (likely) error at first, but the error triggered by the pg_hba failure will trigger the hook to retry, and eventually Patroni will have rendered the pg_hba.conf with the landscape user, and the admin account can be created.

You should be able to go to the IP address of the Landscape Server unit and login as john@example.com with the password pwd.

@jansdhillon jansdhillon requested a review from Copilot March 6, 2026 08:53
@jansdhillon jansdhillon marked this pull request as draft March 6, 2026 08:57
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the charm’s admin account bootstrap flow to intentionally fail (and thus re-trigger Juju hook handling) when the initial bootstrap hits the transient Patroni pg_hba.conf race, and adds unit/integration coverage to exercise the retry behavior with the modern database relation.

Changes:

  • Raise CalledProcessError from _bootstrap_account() when stderr contains no pg_hba.conf entry, enabling hook failure/retry behavior for the transient race.
  • Restructure _migrate_schema_bootstrap() so schema failures are handled distinctly, while bootstrap/autoregistration execute after a successful schema update.
  • Add/repair unit and integration tests to validate bootstrap behavior (including retries) and update an example bundle to configure initial admin credentials.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/charm.py Raises on pg_hba race during bootstrap; adjusts schema/bootstrap sequencing.
tests/unit/test_database_relation.py Adds scenario-based unit tests for bootstrap + pg_hba retry behavior during modern DB relation events.
tests/unit/test_charm.py Re-enables and fixes bootstrap-account unit tests; improves call filtering and mocking paths.
tests/integration/test_bundle.py Adds an integration test asserting the admin account exists after modern DB bootstrap completes.
bundle-examples/internal-haproxy.bundle.yaml Configures admin bootstrap fields in the example bundle for end-to-end verification.

Comment thread tests/unit/test_database_relation.py Outdated
@jansdhillon jansdhillon marked this pull request as ready for review March 6, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants